Note: The information in this document is based on Cisco IOSŪ Software Release 12.0
When a serial line does not come up as it should, the best way to troubleshoot the circuit is to perform loopback testing. Loopback testing allows for pieces of the circuit to be isolated and tested separately. Loopback testing should begin on the customer premises with channel service unit/data service unit (CSU/DSU) loopback testing and then proceed on to loopback testing involving the telco or provider.
There are two kinds of loopback tests available for customers to isolate problems on the serial link: software loopbacks and hardware plug loopbacks. Whether it is an internal or external CSU/DSU, both software and hardware loopbacks can be done back towards the router.
Software local loopbacks are usually implemented with a Cisco IOS configuration command or with a loopback button for some CSU/DSUs.
Hardware loopbacks are done using a loopback plug or cable inserted into the CSU/DSU.
If CSU/DSU loopback tests prove the router equipment, CSU/DSU, and any cables connecting them are not faulty, then further loopback testing needs to be done with the Telco/Provider of the circuit to isolate your issue.
The diagram below describes the various loopback tests that can be done to accurately isolate your serial line issue.
Warning: All loopback testing is intrusive to the circuit. This means that while you are troubleshooting your circuit you will not be able to pass traffic across that link
Note: All loopback testing is done with High-Level Data Link Control (HDLC) encapsulation.
Note: Refer to the "Local Loop to Router " shown in the diagram above.
Although both software and hardware loopback tests can be done on a CSU/DSU, using a loopback plug is more effective to isolate problems. This is because a software loopback back towards the router will usually only loop the DSU functionality of a CSU/DSU, whereas a hardware loopback is able to prove that the entire CSU/DSU is not at fault.
For an internal CSU/DSU, the software loopback is implemented with a Cisco IOS configuration command. For most platforms the command will be of the form loopback or loopback dte or loopback local. This will loop the circuit from inside the CSU/DSU back towards the router, thereby isolating that section of the circuit.
To run the loopback test on channelized T1s using Primary Rate Interface (PRI) or channel associated signaling(CAS), you need to use the channel-group T1 controller command to create one or more serial interfaces mapped to a set of timeslots in the channelized T1. If the T1 is configured as a PRI you need to remove the pri-group before using the channel-group command. If you wish to run a software loopback at the local CSU, configure loopback local in the controller. For more information on these commands refer to the Command Reference Guide. An example of the use of these commands is shown below:
Note: This creates a single Serial0:0 interface (where the first 0 stands for the controller and the second 0 represents the channel-group number) using all 24 timeslots for a total of 1.536Mbps bandwidth. If you are using Super Frame (SF) framing type & alternate mark inversion (AMI) linecoding , then use "speed 56" in the channel-group command, since SF/AMI does not support clear-channel DS0s.Router#configure terminal Enter configuration commands, one per line. End with CNTL/Z. Router(config)#controller t1 0 Router(config-controller)#no pri-group timeslots 1-24 Router(config-controller)#channel-group 0 timeslots 1-24 speed 64 ! -- This automatically creates a single Serial0:0 interface. Router(config-controller)#loopback local ! -- The loopback local command above is only necessary for software loopbacks.
Router(config-controller)#exit
Router(config)#interface serial 0:0 Router(config-if)#encapsulation hdlc ! -- Note: All loopback testing is done with hdlc encapsulation.
Please refer to the Diagnostic Tests while in Loopback section for information on what should be verified while in loopback.
The hardware loopback plug test is used to see if the router and the entire CSU/DSU has any faults. If a router passes a hardware loopback plug test, then the problem exists elsewhere on the line. Refer to the directions below for creating a loopback plug, and then insert the plug into the network (Telco) side of the CSU/DSU.
For the hardware loopback test, first perform the steps described in the software loopback section, except for configuring loopback local on the controller. If you have configured loopback local on the controller, undo it using the no loopback local command before proceeding.
Please refer to the Diagnostic Tests while in Loopback section for information on what should be verified while in loopback.
Note: The pins on an RJ-45 cable plug are numbered from 1 through 8. With the metal pins of the plug facing toward you, pin 1 is the leftmost pin.
The T1 CSU/DSU has a different pinout than the four-wire 56K CSU/DSU. The connector for the T1 CSU/DSU is an RJ-48C. The connector for the four-wire 56k CSU/DSU is an RJ-48S. Both connectors are compatible with RJ-45 plugs. Instructions for creating loopback plugs for each one is shown below:
Complete the following steps to create a loopback plug for a T1 CSU/DSU:
Use wire cutters to cut a working RJ-45 cable that is 5 inches long with a connector attached.
Strip the wires.
Twist the wires from pins 1 and 4 together.
Twist the wires from pins 2 and 5 together.
Leave the rest of the wires alone.
Complete the following steps to create a loopback plug for a 56K CSU/DSU:
Use wire cutters to cut a working RJ-45 cable that is 5 inches long with a connector attached.
Strip the wires.
Twist the wires from pins 1 and 7 together.
Twist the wires from pins 2 and 8 together.
Leave the rest of the wires alone.
Note: Refer to the "Telco Assisted Loop to Router" shown in the diagram above.
If the CSU/DSU testing above is able to rule out a problem with the CSU/DSU, router, and the cable connecting them (for an external CSU/DSU) on both sides of the circuit, then the next step is to involve the Telco/Provider. This loopback testing is done with the Telco's assistance, but it is not done independently by the Telco. This is not the same thing as the Telco performing diagnostic or Bit Error Rate Test (BERT) testing on the line.
For this loopback testing, you will need to involve the Telco since you will be asking for loopbacks to be provided by them towards your premises from the Telco switches. You will then monitor the looped circuit from the router. To do this you need to have the Telco "split the circuit" at the Telco switch closest to your router. For example, Telco should supply a loopback at the the first Telco switch that your circuit passes through, and loop that circuit back towards your router. In this way you can isolate the Telco cloud of switches and only test the piece of the circuit from the first Telco switch down to and including your CSU/DSU, SmartJack, and router.
Please refer to the Diagnostic Tests while in Loopback section for information on what should be verified while in loopback.
Once this "First Switch" testing is completed and is proven to run clean and free of errors by you, perform the same procedure on the remote end of the circuit (the router on the other side of the provider cloud). If the remote end is your Internet service provider (ISP), then you will need to involve the ISP to help test this portion of the circuit.
If the "First Switch" testing is done on both sides and proven to be clean, you can present this information to your telco as an indication that the problem appears to be within the telco cloud. They can either investigate with their own testing of the circuit at this point, or continue the loopback testing with you, backing off one switch at a time further into the telco cloud, doing a loopback towards the local router at each switch.
If the "First Switch" testing proves that there is a problem in the circuit between the first telco switch and your router, the telco can help to test that portion of the circuit. Between the SmartJack that you connect your CSU/DSU to and the first telco switch, the telco has several pieces of equipment that they can loop for diagnostic tests. Also, keep in mind that if you have an extended demarc, this should be investigated as a potential problematic area, since extended demarcs, when done incorrectly, can produce errors on the line. Extended demarcs are when the providers extends the original demarc point to a location closer to the customer equipment.
The best test to run while in any of the loopbacks described above is an extend ping. You should run this test and monitor the show interface serial command for errors on the interface. This test is described below:
Router#show interface serial 0 Serial0 is up, line protocol is up (looped) Hardware is HD64570 Internet address is 10.1.1.1, subnet mask is 255.255.255.0 MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec, rely 255/255, load 1/255 Encapsulation HDLC, loopback set, keepalive set (10 sec) ...
Router(config-if)#ip address 172.22.53.1 255.255.255.0
Router#clear counters Clear "show interface" counters on all interfaces [confirm] Router#
Note:The ping command is particularly useful when high levels of input errors are being registered in the show interfaces serial output.
Cisco internetworking devices provide a mechanism to automate the sending of many ping packets in sequence.
Complete the following steps to perform serial line extended ping tests:
Notice that the ping packet size is 1500 bytes, and that we are performing an all zeros ping (0x0000). Also, the ping count specification is set to 50. Therefore, in this case, there are fifty 1500 byte ping packets sent out.A sample output is below:
Router#ping ip Target IP address: 172.22.53.1 Repeat count [5]: 50 Datagram size [100]: 1500 Timeout in seconds [2]: Extended commands [n]: yes Source address or interface: Type of service [0]: Set DF bit in IP header? [no]: Validate reply data? [no]: Data pattern [0xABCD]: 0x0000 Loose, Strict, Record, Timestamp, Verbose[none]: Sweep range of sizes [n]: Type escape sequence to abort. Sending 50, 1500-byte ICMP Echos to 172.22.53.1, timeout is 2 seconds: Packet has data pattern 0x0000 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Success rate is 100 percent (50/50), round-trip min/avg/max = 4/4/8 ms Router#
Repeat step 1, but use a Data Pattern of 0x1111
Repeat step 1, but use a Data Pattern of 0xffff
Repeat step 1, but use a Data Pattern of 0xaaaa
Verify that all the extended ping tests were 100 percent successful.
Examine the show interface serial command output and determine if input errors have increased. If input errors have not increased, the local hardware (DSU, cable, router interface card) is probably in good condition. Also look for cyclic redundancy check (CRC), frame, or other errors. Verify this by looking at the fifth and sixth line from the bottom of the show interface serial command output.
If all pings are 100 percent successful and input errors have not increased, the equipment in this portion of the circuit is probably in good condition. Move on to the next loopback test to be performed..
Remove the loopback from the interface (by removing the loopback plug, the software loopback commands, or request the telco to remove their loopback) and restore your router to the original setting.